Preventing motion artifacts by intelligently disabling video stabilization

ABSTRACT

Digital video stabilization is selectively turned off in circumstances where it could actually decrease the quality of a captured video. A video camera includes a device for directly detecting physical motion of the camera. Motion data from the motion detector are analyzed to see if video stabilization is appropriate. If the motion data indicate that the video camera is stable, for example, then video stabilization is not applied to the video, thus preventing the possibility of introducing “motion artifacts” into the captured video. In another example, motion as detected by the motion detector can be compared with motion as detected by the video-stabilization engine. If the two motions disagree significantly, then the video-stabilization engine is probably responding more to motion in the captured video rather than to motion of the camera itself, and video stabilization should probably not be applied to the video.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is related to U.S. patent applicationPCT/US13/579,07, filed on an even date herewith.

TECHNICAL FIELD

The present disclosure is related generally to digital-image captureand, more particularly, to digital video stabilization.

BACKGROUND

Digital video-capture devices, such as still and video cameras, arebecoming more commonplace. Once available only on high-end professionalcameras, today such digital devices are found in smartphones, tabletcomputers, and other personal electronics devices. Digital cameras aresmall and relatively inexpensive to manufacture and are thus popularaccessories in today's personal electronics devices.

Because these personal electronics devices are often quite small andlight, they can be easily shaken when the user is trying to take aphotograph or a movie, and this shaking can produce blurry images. Tocompensate for shaking, digital video stabilization methods comparecaptured images frame by frame and apply global motion compensation toremove the frame-to-frame motion and to thus make the captured imagesless blurry.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

While the appended claims set forth the features of the presenttechniques with particularity, these techniques, together with theirobjects and advantages, may be best understood from the followingdetailed description taken in conjunction with the accompanying drawingsof which:

FIG. 1 is an schematic of a representative personal electronics devicewith a video camera in which the present techniques may be practiced;

FIG. 2 is a flowchart of a representative method for disabling videostabilization when a camera is steady;

FIGS. 3a through 3c are graphs of motion-data measurements made for acamera in different circumstances; and

FIGS. 4 and 5 are flowcharts of methods for selectively applying videostabilization when a camera may be moving.

DETAILED DESCRIPTION

Turning to the drawings, wherein like reference numerals refer to likeelements, techniques of the present disclosure are illustrated as beingimplemented in a suitable environment. The following description isbased on embodiments of the claims and should not be taken as limitingthe claims with regard to alternative embodiments that are notexplicitly described herein.

Digital video stabilization is selectively turned off in circumstanceswhere it could actually decrease the quality of a captured video. Avideo camera includes a motion detector device for directly detectingphysical motion of the camera (e.g., a gyroscope, accelerometer, a GPSreceiver, or another camera). Motion data from the motion detector areanalyzed to see if video stabilization is appropriate. If the motiondata indicate that the video camera is stable, for example, then videostabilization is not applied to the video, thus preventing thepossibility of introducing “motion artifacts” into the captured video.In another example, motion as detected by the motion detector can becompared with motion as detected by the video-stabilization engine. Ifthe two motions disagree significantly, then the video-stabilizationengine is probably responding more strongly to motion in the capturedimages than to motion of the camera itself, and video stabilizationshould probably not be applied to the video.

Various embodiments can be implemented in hardware or as executable coderunning on a microprocessor or a controller of an video-capture system.Where so configured, the executable code can include various modulesthat determine video stability and physical motion.

FIG. 1 shows a representative personal electronics device 100 that canembody aspects of the present disclosure. The device 100 can be justabout any device with a video camera 102 including, for example, adedicated camera, a mobile telephone, a personal digital assistant, or atablet computer.

The video camera 102 of FIG. 1 represents an entire video-capturedevice, as known in the art. A typical video-capture device 102 includesan image sensor and a control circuit (represented in FIG. 1 by theprocessor 106, but see the discussion below). The image sensor, whichcan be any type of image sensor known in the art, such as a CMOS or aCCD sensor, receives light through a lens. The light is reflected off asubject and is then captured by the image sensor to form an image. Thecaptured image is stored as one digital frame 104. Generally speaking,the video camera 102 creates frames of captured images as a constantrate. When displayed to a user at the same rate, the succession offrames is a video clip. As is known in the art, the video-capture device102 may include other optical, mechanical, and electronic systems(including a variable aperture, lenses, an auto-focus engine, and thelike). The personal electronics device 100 can include another videocamera (not shown in FIG. 1), the possible use of which is discussedbelow.

For ease of discussion, it is assumed that the personal electronicsdevice 100 is completely controlled by a software-driven processor 106.In some embodiments, however, a microcontroller, embedded controller, orapplication-specific controller 106 controls certain aspects of thedevice 100 including the video stabilizer 108 (also called avideo-stabilization engine) and the motion detector 110 as describedbelow in reference to FIGS. 2, 4, and 5. In some embodiments, controlfunctions are divided among multiple modules within the device 100including both software-controlled processors and hardwired controllers.

The video-stabilization engine 108, as is well known in the art, reviewsthe captured frames 104 and attempts to detect motion of the personalelectronics device 100. If, for example, a pixel-by-pixel comparison oftwo successive frames 104 reveals that the second frame is essentiallyidentical to the first but displaced by a small but constant amount tothe right, then it is possible that the image being captured has notmoved to the right, but instead the device 100 has moved that smalldistance to the left. This often happens when the hands holding thedevice 100 shake. Continuing with the above example, in traditionaloperation, the video-stabilization engine 108 attempts to compensate forthis small movement of the device 100 by shifting the image of thesecond frame back to the left. The motion of the device 100 has beencompensated for by moving the image in the succession of captured frames104, and in the resultant video, which comprises the motion-compensatedoutput frames 112, there are, ideally, no lingering effects of theshaking of the device 100.

(As a side note, in order to shift the captured image to compensate forthe slight motion of the personal electronics device 100, the capturedframes 104 are actually somewhat larger than the video eventuallydisplayed. The captured frames 104 include a small extra border ofpixels so that if, for example, the image has shifted to the left andthe leftmost pixels have thus disappeared off the edge of the normalimage, these leftmost pixels are still captured in the left border andcan be used when the image is moved back to the right to compensate forthe motion. This is all well known in the art.)

The motion detector module 110 directly detects physical motion of thepersonal electronics device 100. Various known hardware implementationsare possible here. A three-axis accelerometer can report movement, ascan a gyroscope, or a GPS receiver. In some embodiments, the motiondetector module 110 is based on an image sensor (that is, an imagesensor other than the video camera 102, e.g., a second camera), and itindirectly calculates motion based on perceived motion.

Generally speaking, the personal electronics device 100 includes othercomponents such as a screen for viewing captured images and video, aninterface for receiving user commands and for displaying statusinformation, memory for storing controller software and captured frames,and a communications module for sending and receiving information.Because these components are well known in the art and are not ofparticular concern for the present discussion, they need not bediscussed further.

FIG. 2 shows a first representative method usable by the personalelectronics device 100 of FIG. 1. In step 200, the video-capture engine(or camera) 102 captures and stores a succession of image frames 104.

(Note that for ease of discussion, the representative methods of FIGS.2, 4, and 5 are illustrated and discussed sequentially. In manyembodiments, various steps are performed concurrently, possibly bydifferent processors and controllers working, at least to some extent,independently.)

In step 202, well known techniques of video-stabilization analysis areapplied to the succession of captured frames 104. In one embodiment,successive frames are compared pixel-by-pixel by the video-stabilizationengine 108. If the analysis decides that the differences between framesare best explained, at least in part, by motion of the personalelectronics device 100, then one or more stabilization-compensationvalues are calculated. In some well known embodiments of videostabilization, a calculated stabilization-compensation value minimizes asum of “pixel-error” differences if it were applied to the second of twosuccessive captured frames 104. For side-to-side motion, for example, astabilization-compensation value could be a vector whose scalar value isthe number of pixels that the detected motion represents and thereforethe number of pixels to shift the second image to match (at leastapproximately) the first image to compensate for motion of the device100. In a general case, the analysis can detect motion in all threespatial dimensions as well as rotations. (Note that in step 202, thestabilization-compensation values are calculated but are not applied tothe captured frames 104.)

In many embodiments, the motion detector 110 constantly produces motiondata that represent the detected motions of the personal electronicsdevice 100. In other embodiments, motion data are only produced ifmotion is detected (i.e., no updated motion data mean no detectedmotion).

In any case, the current motion data from step 204 are analyzed in step206. Specifically, the currently detected motion data are comparedagainst a noise threshold value. Most motion detectors 110 known in theart produce some statistically random motion data even when the personalelectronics device 100 is completely at rest. Laboratory tests can berun for a particular motion detector 110 by locking it in place and thenreading its motion-data output. Viewing the output generally clearlyreveals the noise threshold of the motion detector 110. (Of course,statistical techniques more sophisticated than eyeballing a graph can beused to characterize the noise threshold.)

FIG. 3a shows the results of an actual test of a motion detector 110that was held completely static. The graph of FIG. 3a shows the motiondata (in three dimensions) produced in this static test case. The graphshows very frequent motion events, all of small amplitude. This ischaracteristic of detector noise. For this particular motion detector110, a review of the graph reveals a noise threshold of plus or minus0.08. (The actual value of the noise threshold for a particular motiondetector 110 is of less importance than the presence of a well definedthreshold. Of course, lower threshold values are generally better, butachieving a very low threshold may involve production costs that faroutweigh any added benefits.)

If the motion data produced in step 204 are below the experimentallyderived noise threshold of the motion detector 110, then the personalelectronics device 100 is most likely at rest. This is the case even ifthe video-stabilization engine 108 produces a non-zerostabilization-compensation value in step 202. That non-zero value ismost likely to be simply incorrect. Therefore, the method of FIG. 2concludes in step 208 by only applying the stabilization-compensationvalue of step 202 to the captured frames 104 if the motion data producedin step 204 are above the noise threshold of the motion detector 110.Otherwise, the device 100 is static, and the stabilization-compensationvalue is simply ignored.

(The question can be asked: Why would a properly functioningvideo-stabilization engine 108 produce a non-zerostabilization-compensation value when the personal electronics device100 is completely static? The answer is based on the realization thatthe video-stabilization engine 108 does not measure motion directly butonly infers motion by comparing successive captured frames 104. Thatanalysis is “fooled” when, to give just one example, more than half ofthe captured image is actually moving smoothly and steadily across theframe (e.g., when a large object near the front of the scene is movingfrom right to left). In this case, most embodiments of thevideo-stabilization engine 108 believe that it is the device 100 that ismoving rather than the majority of image being captured. Thus, itproduces a non-zero stabilization compensation value when it should, inan ideal world, produce a zero value. If that non-zero value wereapplied to the captured frames 104 to compensate for the non-existentmotion of the static device 100 (that is, if the method of FIG. 2 werenot applied), then the resulting output video 112 would include annoying“motion artifacts.” The method of FIG. 2 prevents these artifacts.)

Note: Performing step 202 consumes power and resources within thepersonal electronics device 100 even when the stabilization-compensationvalues are eventually discarded in step 208 because the device 100 isstatic. Thus, in some embodiments, step 202 is only performed if theanalysis of step 206 reveals that the device 100 may actually be moving.In other embodiments, step 202 is always performed, but the results ofthat step are only used when they are appropriate.

Compare FIG. 3a with FIG. 3b . (The bold horizontal bars at about plusand minus 0.1 of FIG. 3b correspond to the noise threshold measured inFIG. 3a .) The motion data of FIG. 3b were collected while the personalelectronics device 100 was held in one hand. FIG. 3b shows very frequentmotion events, many of which have amplitudes that far exceed the noisethreshold. In this case, video stabilization would be very useful, sostep 208 of FIG. 2 allows the stabilization-compensation values to beapplied to the captured frames 104 to create the output frames 112.(Techniques for applying stabilization-compensation values to video arewell known in the art.)

The data for FIG. 3c were taken when the personal electronics device 100was held in two hands. Comparing the amplitudes and the frequency of themotion events in this figure with those of FIG. 3b , it is clear that atwo-hand grip makes for a much more stable device 100. However, themotion amplitudes in FIG. 3c do still sometimes exceed the noisethreshold, though usually by less than twice the threshold value. Astrict application of the method of FIG. 2 to this situation would allowthe stabilization-compensation values to be applied to the capturedframes 104. However, this is a borderline case, and more data would beuseful to make the best decision about using video stabilization. FIG.4, discussed below, presents one way to do this.

FIG. 4 shows a second representative method usable by the personalelectronics device 100 of FIG. 1. The method of FIG. 4 begins in afashion similar to the method of FIG. 2. Image frames 104 are capturedby the camera 102 (step 400). From the captured frames 104, the videostabilization engine 108 calculates one or morestabilization-compensation values (step 402). Meanwhile, the motiondetector 110 produces motion data (step 404).

In step 406, the method of FIG. 4 diverges from that of FIG. 2. Here,the motion data produced by the motion detector 110 are compared againstthe stabilization-compensation values calculated by the videostabilization engine 108. In high-level terms, this means that twoindependently derived values for the current motion of the personalelectronics device 100 are compared. If the motion detector 110 agrees(within a motion-difference threshold) with the video-stabilizationengine 108 as to how, and how much, the device 100 is currently moving,then their agreed upon value is most likely correct. (Note that, asmentioned above, the motion detector 110 may be implemented as a secondvideo camera, such as a front-facing camera on a laptop computer. Inthis case, step 406 compares the stabilization-compensation values fromthe two cameras. If the value agree within the motion-differencethreshold, then their agreed motion value is most likely to be correct.)In this case, step 408 applies the stabilization-compensation value tothe captured frames 104 to create the output frames 112. If the twomotion values disagree, then the video-stabilization engine 108 isprobably being fooled, so its stabilization-compensation values cannotbe relied upon and are discarded.

Laboratory tests of the personal electronics device 100 can be used toset the value of the motion-difference threshold. In some cases, thenoise threshold derived for the motion detector 110 when the device 100is held static (see FIG. 3a and the accompanying discussion) may be usedfor the motion-difference threshold. It is also conceivable thatlaboratory testing can find, for some hardware embodiments, an optimalmotion-difference threshold that is different from the noise threshold.Different hardware embodiments can lead to different results.

Finally, FIG. 5 shows a third representative method usable by thepersonal electronics device 100 of FIG. 1. This method combines aspectsof the methods of FIGS. 2 and 4. As in those methods, image frames 104are captured by the camera 102 (step 500). From the captured frames 104,the video stabilization engine 108 calculates one or morestabilization-compensation values (step 502). Meanwhile, the motiondetector 110 produces motion data (step 504). As in step 406 of FIG. 4,in step 506 the motion data produced by the motion detector 110 arecompared against the stabilization-compensation values calculated by thevideo stabilization engine 108.

Step 508 applies both the test of step 208 of FIG. 2 and the test ofstep 408 of FIG. 4. Thus, only if the personal electronics device 100 isnot static (that is, if the motion data produced in step 504 are abovethe noise threshold of the motion detector 110), and if the motiondetector 110 agrees (within the motion-difference threshold) with thevideo-stabilization engine 108 as to how, and how much, the device 100is currently moving, then the stabilization-compensation values areapplied to the captured frames 104 to create the output frames 112.Otherwise, the video-stabilization engine 108 is probably being fooled,so its stabilization-compensation values cannot be relied upon and arediscarded.

In view of the many possible embodiments to which the principles of thepresent discussion may be applied, it should be recognized that theembodiments described herein with respect to the drawing figures aremeant to be illustrative only and should not be taken as limiting thescope of the claims. Therefore, the techniques as described hereincontemplate all such embodiments as may come within the scope of thefollowing claims and equivalents thereof.

We claim:
 1. On a personal electronics device comprising a camera, adigital video stabilizer, and a motion detector, a method comprising:acquiring by the camera, a series of images; calculating, based on acomparison of a plurality of images in the series of images by thedigital video stabilizer, a stabilization-compensation value; producing,by the motion detector during the acquiring, motion data; and in theevent that an amount of motion of the personal electronics deviceindicated by the motion data differs from an amount of motion indicatedby the stabilization-compensation value by less than a motion-differencethreshold, applying the stabilization-compensation value to the seriesof images.
 2. The method of claim 1, wherein the personal electronicsdevice includes at least one of a camera, a mobile telephone, a personaldigital assistant, and a tablet computer.
 3. The method of claim 1,wherein the motion detector includes at least one of a gyroscope, anaccelerometer, a GPS receiver, and an image sensor.
 4. The method ofclaim 1, wherein calculating the stabilization-compensation valueincludes minimizing a sum of pixel-error differences between comparedimages of the plurality of images in the series of images.
 5. The methodof claim 1, wherein the stabilization-compensation value includes atleast one of a linear offset and a rotation.
 6. The method of claim 1,wherein calculating the motion-difference threshold comprises measuringan amount of noise produced by the motion detector.
 7. The method ofclaim 1, wherein applying the stabilization-compensation value includesmodifying an image in the series of images, the modifying based, atleast in part, on the stabilization-compensation value, and sending themodified image to a video encoder.
 8. A personal electronics devicecomprising: a camera configured to acquire a series of images; a digitalvideo stabilizer configured to calculate, based on a comparison of aplurality of images in the series of images, astabilization-compensation value; a motion detector configured toproduce motion data while acquiring the series of images; and aprocessor operatively connected to the camera, to the digital videostabilizer, and to the motion detector, the processor configured to: inthe event that an amount of motion of the personal electronics deviceindicated by the motion data differs from an amount of motion indicatedby the stabilization-compensation value by less than a motion-differencethreshold, apply the stabilization-compensation value to the series ofimages.
 9. The personal electronics device of claim 8, wherein thepersonal electronics device includes at least one of a camera, a mobiletelephone, a personal digital assistant, and a tablet computer.
 10. Thepersonal electronics device of claim 8, wherein the motion detectorincludes at least one of a gyroscope, an accelerometer, a GPS receiver,and an image sensor.
 11. The personal electronics device of claim 8,wherein calculating the stabilization-compensation value includesminimizing a sum of pixel-error differences between compared images ofthe plurality of images in the series of images.
 12. The personalelectronics device of claim 8, wherein the stabilization-compensationvalue includes at least one of a linear offset and a rotation.
 13. Thepersonal electronics device of claim 8, wherein calculating themotion-difference threshold comprises measuring an amount of noiseproduced by the motion detector.
 14. The personal electronics device ofclaim 8, wherein applying the stabilization-compensation value includesmodifying an image in the series of images, the modifying based, atleast in part, on the stabilization-compensation value, and sending themodified image to a video encoder.
 15. A personal electronics devicecomprising: a camera configured to acquire a series of images; a digitalvideo stabilizer configured to calculate, based on a comparison of aplurality of images in the series of images, astabilization-compensation value; a motion detector configured toproduce motion data while acquiring the series of images, the producingmotion data being separate from the calculating thestabilization-compensation value; and a processor operatively connectedto the camera, the digital video stabilizer and the motion detector, theprocessor configured to: determine a difference between an amount ofmotion indicated by the motion data and an amount of motion indicated bythe stabilization-compensation value; and if the difference is less thana motion-difference threshold, apply the stabilization-compensationvalue to the series of images.
 16. The personal electronics device ofclaim 15, wherein the personal electronics device includes at least oneof a camera, a mobile telephone, a personal digital assistant, and atablet computer.
 17. The personal electronics device of claim 15,wherein the motion detector includes at least one of a gyroscope, anaccelerometer, a GPS receiver, and an image sensor.